Spatial Temporal Graph Convolutional Network Research Articles

Short-term load forecasting of regional integrated energy system based on spatio-temporal convolutional graph neural network

Accurate short-term load forecasting has significant reference value for the preliminary planning and subsequent coordinated control of regional integrated energy systems. However, main limitations are: (1) Insufficient exploration of the complex correlations among diverse loads and between loads and meteorological factors; and (2) Feature information extraction only from the time dimension, neglecting the spatial dimension, resulting in incomplete feature information extraction. This paper proposes a new method for short-term load forecasting, namely the Attention Spatial–temporal Graph Convolutional Network (ATSTGCN) method. Firstly, the Grey Relational Analysis (GRA) method is used to analyze the correlation between diverse loads, meteorological features, and load features, constructing the input data for the graph neural network. Then, synchronous graph convolution is introduced to extract information between features. Next, the spatio-temporal attention mechanism is incorporated into the synchronous graph convolution to capture spatio-temporal features and reduce information loss using a spatio-temporal window. Finally, a short-term load forecasting method based on ATSTGCN for regional integrated energy systems is constructed. Extensive simulation experiments are conducted on the publicly available dataset from Arizona State University’s online system. The experimental results show that compared to other methods, the proposed method improves MAPE, MAE, and RMSE by 5.6%, 11.7%, and 11.6%.

Dynamic multi-scale spatial–temporal graph convolutional network for traffic flow prediction

As a critical component of Intelligent Transportation Systems (ITS), traffic flow prediction is indispensable for vehicle routing and transportation management. However, traffic prediction is an intractable task due to the complex spatial–temporal dependencies. First, traffic data has multiple spatial dependencies that arise from different contexts and vary with time. Second, multi-scale temporal features naturally exist in the traffic data and have dynamic significance for predicting future traffic states. Insufficiently capturing such spatial and temporal dependencies will lead to the prediction performance degradation. To this end, we propose a dynamic multi-scale spatial–temporal graph convolutional network for traffic flow prediction. Unlike existing methods, we propose a novel attention layer assigning weights to each feature of nodes to augment the dynamic spatial dependence modeling and a dynamic multi-branch temporal network learning temporal features from the time domain and frequency domain complementary to enhance the dynamic multi-scale temporal features learning capability. Extensive experiments on four well-known traffic datasets demonstrate that the proposed model outperforms the best baselines with a ratio of 5.27%, 2.44%, 3.27%, and 7.34% in terms of mean absolute error and also has superior long-term prediction performance than state-of-the-art baselines.

Human Action Recognition and Note Recognition: A Deep Learning Approach Using STA-GCN.

Human action recognition (HAR) is growing in machine learning with a wide range of applications. One challenging aspect of HAR is recognizing human actions while playing music, further complicated by the need to recognize the musical notes being played. This paper proposes a deep learning-based method for simultaneous HAR and musical note recognition in music performances. We conducted experiments on Morin khuur performances, a traditional Mongolian instrument. The proposed method consists of two stages. First, we created a new dataset of Morin khuur performances. We used motion capture systems and depth sensors to collect data that includes hand keypoints, instrument segmentation information, and detailed movement information. We then analyzed RGB images, depth images, and motion data to determine which type of data provides the most valuable features for recognizing actions and notes in music performances. The second stage utilizes a Spatial Temporal Attention Graph Convolutional Network (STA-GCN) to recognize musical notes as continuous gestures. The STA-GCN model is designed to learn the relationships between hand keypoints and instrument segmentation information, which are crucial for accurate recognition. Evaluation on our dataset demonstrates that our model outperforms the traditional ST-GCN model, achieving an accuracy of 81.4%.

Multimodal fall detection for solitary individuals based on audio-video decision fusion processing

Falls often pose significant safety risks to solitary individuals, especially the elderly. Implementing a fast and efficient fall detection system is an effective strategy to address this hidden danger. We propose a multimodal method based on audio and video. On the basis of using non-intrusive equipment, it reduces to a certain extent the false negative situation that the most commonly used video-based methods may face due to insufficient lighting conditions, exceeding the monitoring range, etc. Therefore, in the foreseeable future, methods based on audio and video fusion are expected to become the best solution for fall detection. Specifically, this article outlines the following methodology: the video-based model utilizes YOLOv7-Pose to extract key skeleton joints, which are then fed into a two stream Spatial Temporal Graph Convolutional Network (ST-GCN) for classification. Meanwhile, the audio-based model employs log-scaled mel spectrograms to capture different features, which are processed through the MobileNetV2 architecture for detection. The final decision fusion of the two results is achieved through linear weighting and Dempster-Shafer (D-S) theory. After evaluation, our multimodal fall detection method significantly outperforms the single modality method, especially the evaluation metric sensitivity increased from 81.67% in single video modality to 96.67% (linear weighting) and 97.50% (D-S theory), which emphasizing the effectiveness of integrating video and audio data to achieve more powerful and reliable fall detection in complex and diverse daily life environments.

Deep learning-based spatial-temporal graph neural networks for price movement classification in crude oil and precious metal markets

In this study, we adapt three spatial-temporal graph neural network models to the unique characteristics of crude oil, gold, and silver markets for forecasting purposes. It aims to be the first to (i) explore the potential of spatial-temporal graph neural networks family for price forecasting of these markets, (ii) examine the role of attention mechanism in improving forecasting accuracy, and (iii) integrate various sources of predictors for better performance. Specifically, we present three distinct models: Multivariate Time Series Graph Neural Networks with Temporal Attention and Learnable Adjacency matrix (MTGNN-TAttLA), Spatial Attention Graph with Temporal Convolutional Networks (SAG-TCN), and Attention-based Spatial-Temporal Graph Convolutional Networks (ASTGCN), to capture the intricate interplay of spatial and temporal dependencies within crude oil and precious metals markets. Moreover, the effectiveness of the attention mechanism in improving models' accuracies is shown. Our empirical results reveal remarkable prediction accuracy, with all three models outperforming conventional deep learning methods such as Temporal Convolutional Networks (TCN), long short-term memory networks (LSTM) and convolutional neural networks (CNN). The MTGNN-TAttLA model, enriched with a temporal attention mechanism, exhibits exceptional performance in predicting the direction of price movement in the WTI, Brent, and silver markets, while ASTGCN is the best-performing model for the gold market. Additionally, we observed that incorporating technical indicators from the crude oil and precious metal markets into the graph structure has improved the classification accuracy of spatial-temporal graph neural networks.

LightPRA: A Lightweight Temporal Convolutional Network for Automatic Physical Rehabilitation Exercise Assessment

Research evidence shows that physical rehabilitation exercises prescribed by medical experts can assist in restoring physical function, improving life quality, and promoting independence for physically disabled individuals. In response to the absence of immediate expert feedback on performed actions, developing a Human Action Evaluation (HAE) system emerges as a valuable automated solution, addressing the need for accurate assessment of exercises and guidance during physical rehabilitation. Previous HAE systems developed for the rehabilitation exercises have focused on developing models that utilize skeleton data as input to compute a quality score for each action performed by the patient. However, existing studies have focused on improving scoring performance while often overlooking computational efficiency. In this research, we propose LightPRA (Light Physical Rehabilitation Assessment) system, an innovative architectural solution based on a Temporal Convolutional Network (TCN), which harnesses the capabilities of dilated causal Convolutional Neural Networks (CNNs). This approach efficiently captures complex temporal features and characteristics of the skeleton data with lower computational complexity, making it suitable for real-time feedback provided on resource-constrained devices such as Internet of Things (IoT) devices and Edge computing frameworks. Through empirical analysis performed on the University of Idaho-Physical Rehabilitation Movement Data (UI-PRMD) and KInematic assessment of MOvement for remote monitoring of physical REhabilitation (KIMORE) datasets, our proposed LightPRA model demonstrates superior performance over several state-of-the-art approaches such as Spatial–Temporal Graph Convolutional Network (STGCN) and Long Short-Term Memory (LSTM)-based models in scoring human activity performance, while exhibiting lower computational cost and complexity.

A motion-aware and temporal-enhanced Spatial–Temporal Graph Convolutional Network for skeleton-based human action segmentation

Action segmentation task is an important approach for understanding the actions from the video. Most of the conventional action recognition tasks can recognize only a single action from a given input video, thus we need to input a pre-trimmed video containing only one type of action. In contrast, temporal action segmentation (TAS) aims to segment a temporally untrimmed video sequence by time. Consequently, it has wider application prospects in various fields. Previously proposed TAS-based methods use only RGB color video as input to segment the actions, but RGB video is not robust against diverse backgrounds. Whereas skeleton-based features are more resilient as they do not incorporate any background information but there has been limited research exploring this feature modality. To this end, we propose a motion-aware and temporal-enhanced spatial–temporal graph convolutional network for the skeleton-based human action segmentation. Our framework contains a motion-aware module, multi-scale temporal convolutional network, temporal-enhanced graph convolutional network module and a refinement module. Our method can efficiently capture the motion information and long-range dependencies using skeleton features while improving temporal modeling. We have conducted experiments using four publicly available datasets to demonstrate the effectiveness of our introduced method. The code is available at https://github.com/11yxk/openpack.

W-WaveNet: A multi-site water quality prediction model incorporating adaptive graph convolution and CNN-LSTM.

Water quality prediction is of great significance in pollution control, prevention, and management. Deep learning models have been applied to water quality prediction in many recent studies. However, most existing deep learning models for water quality prediction are used for single-site data, only considering the time dependency of water quality data and ignoring the spatial correlation among multi-sites. This research defines and analyzes the non-aligned spatial correlations that exist in multi-site water quality data. Then deploy spatial-temporal graph convolution to process water quality data, which takes into account both the temporal and spatial correlation of multi-site water quality data. A multi-site water pollution prediction method called W-WaveNet is proposed that integrates adaptive graph convolution and Convolutional Neural Network, Long Short-Term Memory (CNN-LSTM). It integrates temporal and spatial models by interleaved stacking. Theoretical analysis shows that the method can deal with non-aligned spatial correlations in different time spans, which is suitable for water quality data processing. The model validates water quality data generated on two real river sections that have multiple sites. The experimental results were compared with the results of Support Vector Regression, CNN-LSTM, and Spatial-Temporal Graph Convolutional Networks (STGCN). It shows that when W-WaveNet predicts water quality over two river sections, the average Mean Absolute Error is 0.264, which is 45.2% lower than the commonly used CNN-LSTM model and 23.8% lower than the STGCN. The comparison experiments also demonstrate that W-WaveNet has a more stable performance in predicting longer sequences.

Spatial-temporal graph convolution network model with traffic fundamental diagram information informed for network traffic flow prediction

Accurate and fine-grained traffic state prediction has always been an important research field. For long-term traffic flow prediction, the high-dimensional and coupled traffic feature evolution patterns are deeply recessive, posing challenges in effectively characterizing and modeling them. This paper proposed a novel spatial–temporal graph convolution network model with traffic Fundamental Diagram (FD) information informed. The model decouples the high-dimensional spatiotemporal relationships in the transportation network and fully leverages the physical evolution laws of traffic states. First, the Graph Convolutional Network (GCN) with a spatial attention mechanism was proposed to capture spatial relations of the road network. The mechanism can better represent the spatial dynamics of the graph adjacency matrix in GCN. Second, this study injected prior physical knowledge into the graph adjacency matrix. This process was achieved by embedding characteristics of FDs from historical traffic data on the diagonal of the matrix, by which the propagation pattern of traffic states in road network could be considered. Third, to further catch the time dependence of the road network, the Gated Recurrent Unit (GRU) structure and the Transformer encoding structure were employed to locally and globally reform traffic state time sequences. Finally, experiments on a revised traffic dataset demonstrated that the proposed method consistently outperforms other baselines regarding Mean Absolute Error and Root Mean Square Error across all cases. Moreover, it achieved the optimal Mean Absolute Percentage Error in the 30- and 60-minute prediction tasks. This study shows a novel solution to inform traffic physical laws into data-driven state prediction models, and the reliability of the proposed method in long-term prediction offers valuable support for improving traffic management and alleviating traffic congestion.

Skeleton-Based Activity Recognition for Process-Based Quality Control of Concealed Work via Spatial-Temporal Graph Convolutional Networks.

Computer vision (CV)-based recognition approaches have accelerated the automation of safety and progress monitoring on construction sites. However, limited studies have explored its application in process-based quality control of construction works, especially for concealed work. In this study, a framework is developed to facilitate process-based quality control utilizing Spatial-Temporal Graph Convolutional Networks (ST-GCNs). To test this model experimentally, we used an on-site collected plastering work video dataset to recognize construction activities. An ST-GCN model was constructed to identify the four primary activities in plastering works, which attained 99.48% accuracy on the validation set. Then, the ST-GCN model was employed to recognize the activities of three extra videos, which represented a process with four activities in the correct order, a process without the activity of fiberglass mesh covering, and a process with four activities but in the wrong order, respectively. The results indicated that activity order could be clearly withdrawn from the activity recognition result of the model. Hence, it was convenient to judge whether key activities were missing or in the wrong order. This study has identified a promising framework that has the potential to the development of active, real-time, process-based quality control at construction sites.

Local-Global Spatial-Temporal Graph Convolutional Network for Traffic Flow Forecasting

Traffic forecasting’s key challenge is to extract dynamic spatial-temporal features within intricate traffic systems. This paper introduces a novel framework for traffic prediction, named Local-Global Spatial-Temporal Graph Convolutional Network (LGSTGCN). The framework consists of three core components. Firstly, a graph attention residual network layer is proposed to capture global spatial dependencies by evaluating traffic mode correlations between different nodes. The context information added in the residual connection can improve the generalization ability of the model. Secondly, a T-GCN module, combining a Graph Convolution Network (GCN) with a Gated Recurrent Unit (GRU), is introduced to capture real-time local spatial-temporal dependencies. Finally, a transformer layer is designed to extract long-term temporal dependence and to identify the sequence characteristics of traffic data through positional encoding. Experiments conducted on four real traffic datasets validate the forecasting performance of the LGSTGCN model. The results demonstrate that LGSTGCN can achieve good performance and be applicable to traffic forecasting tasks.

Compensation Method for Missing and Misidentified Skeletons in Nursing Care Action Assessment by Improving Spatial Temporal Graph Convolutional Networks.

With the increasing aging population, nursing care providers have been facing a substantial risk of work-related musculoskeletal disorders (WMSDs). Visual-based pose estimation methods, like OpenPose, are commonly used for ergonomic posture risk assessment. However, these methods face difficulty when identifying overlapping and interactive nursing tasks, resulting in missing and misidentified skeletons. To address this, we propose a skeleton compensation method using improved spatial temporal graph convolutional networks (ST-GCN), which integrates kinematic chain and action features to assess skeleton integrity and compensate for it. The results verified the effectiveness of our approach in optimizing skeletal loss and misidentification in nursing care tasks, leading to improved accuracy in calculating both skeleton joint angles and REBA scores. Moreover, comparative analysis against other skeleton compensation methods demonstrated the superior performance of our approach, achieving an 87.34% REBA accuracy score. Collectively, our method might hold promising potential for optimizing the skeleton loss and misidentification in nursing care tasks.

A coarse-to-fine adaptive spatial–temporal graph convolution network with residuals for motor imagery decoding from the same limb

In the field of Brain Computer Interface (BCI) technology, Motor Imagery (MI) plays an important role as a paradigm. One of the primary focuses of this research area lies in exploring the MI of various upper limbs. Decoding MI signals from distinct joints within the same limb poses more intricate challenges in comparison to decoding MI signals originating from different upper limbs. In order to explore more efficient decoding methods, we propose a novel coarse-to-fine classification approach to investigate categorical decoding across three tasks, namely ‘rest’, ‘hand’, and ‘elbow’. This approach consists of two classification stages performed from coarse to fine. In the coarse classification stage, in order to capture the features of both resting state and the moving state in the temporal domain, the EEGNet network with temporal domain convolution is used to extract temporal domain features and classify the original samples into categories of ‘rest’ and ‘move’ (‘hand’, ‘elbow’). In the fine classification stage, the samples of ‘move’ category are segmented by time to form a graph sequence. Then, an adaptive spatial–temporal graph convolutional network with residuals is utilized to extract both temporal and spatial domains’ features from the graph sequence. The proposed algorithm has been validated experimentally on the MI-2 dataset and compared with contemporary methods. Its classification performance is quantified by the average accuracy which achieves a value of 72.21%. Extensive experimental results indicate that the novel coarse-to-fine classification approach is superior to the single classification approach.

An attention-based adaptive spatial–temporal graph convolutional network for long-video ergonomic risk assessment

Ergonomic risk assessment (ERA) is commonly used to identify and analyze postures that are detrimental to the health of workers in industrial workplaces, which is vital to prevent work-related musculoskeletal disorders (WMSDs). Among the automatic approaches, algorithms based on graph convolutional networks (GCNs) have shown promising results in ERA using skeleton sequence as input. However, previous GCN-based methods still have certain limitations. First, the separated modeling of spatial and temporal information and the manually pre-defined topology of graph may restrict the representation diversity of the networks. Additionally, RNN-based temporal modeling often incurs high computational costs and fails to capture long-range temporal dependencies, thereby reducing flexibility in describing long videos. To overcome these challenges, in this study, we propose an attention-based adaptive spatial–temporal graph convolutional network (AAST-GCN), aiming to achieve effective and efficient action representation for ERA in long video. First, we employ an alternate modeling strategy to effectively capture the spatial–temporal information, and propose an improved adaptive adjacency matrix scheme to learn various coordination and relations of body-joints, thus enhancing the flexibility to model diverse postures. Furthermore, we introduce an efficient multi-scale temporal convolutional network as a replacement for RNN-based algorithms, enabling the network to extract various granularities of temporal features. Moreover, to make the network focuses on more valuable information, we employ a spatial–temporal interaction attention (STIA) module. Finally, the aforementioned modules are aggregated within a multi-task learning framework, with the action segmentation serving as the auxiliary task to further improve the accuracy of ERA. We conducted the ergonomic risk assessment on the UW-IOM and TUM Kitchen datasets using our network. Extensive experiments conducted on the most popular datasets UW-IOM and TUM Kitchen demonstrated that our proposed AAST-GCN outperforms other GCN-based methods. Ablation studies and visualization also prove the effectiveness of the individual sub-modules.

Wind Power Forecasting Based on a Spatial–Temporal Graph Convolution Network With Limited Engineering Knowledge

Wind Power Forecasting Based on a Spatial–Temporal Graph Convolution Network With Limited Engineering Knowledge

Skeleton-Based Action Segmentation With Multi-Stage Spatial-Temporal Graph Convolutional Neural Networks

The ability to identify and temporally segment fine-grained actions in motion capture sequences is crucial for applications in human movement analysis. Motion capture is typically performed with optical or inertial measurement systems, which encode human movement as a time series of human joint locations and orientations or their higher-order representations. State-of-the-art action segmentation approaches use multiple stages of temporal convolutions. The main idea is to generate an initial prediction with several layers of temporal convolutions and refine these predictions over multiple stages, also with temporal convolutions. Although these approaches capture long-term temporal patterns, the initial predictions do not adequately consider the spatial hierarchy among the human joints. To address this limitation, we recently introduced multi-stage spatial-temporal graph convolutional neural networks (MS-GCN). Our framework replaces the initial stage of temporal convolutions with spatial graph convolutions and dilated temporal convolutions, which better exploit the spatial configuration of the joints and their long-term temporal dynamics. Our framework was compared to four strong baselines on five tasks. Experimental results demonstrate that our framework is a strong baseline for skeleton-based action segmentation.

Cross-User Electromyography Pattern Recognition Based on a Novel Spatial-Temporal Graph Convolutional Network

Cross-User Electromyography Pattern Recognition Based on a Novel Spatial-Temporal Graph Convolutional Network

Motion Complement and Temporal Multifocusing for Skeleton-Based Action Recognition

Modeling sequences with spatial-temporal graph convolutional networks has become a mainstream paradigm in skeleton-based action recognition. However, many existing methods adopt redundant or cluttered structures to mine the key action features, thus making it difficult to achieve a balanced or leading performance in accuracy and efficiency. In this paper, we propose a novel framework, referred to as Motion Complement and Temporal Multifocusing Network (MCTM-Net), to capture the relationships within skeleton sequences by means of an efficient decomposition of the spatiotemporal graph model. Specifically, for spatial modeling, we introduce a motion-related relational descriptor that extends the channel dimension so as to enhance the modeling of motion salient regions as a complement to the conventional physical adjacency relationships. An improved parameterized physical relationship model is also proposed to better fit the data characteristics. As for temporal modeling, we propose an efficient multi-focus temporal information acquisition strategy that aggregates the information from multiple temporal spans and adjacent regions. We conduct extensive experiments on multiple representative datasets, including NTU-RGB+D (60&120), Northwestern-UCLA, and UWA3D Multiview Activity II, to validate our innovations. The experimental results show the effectiveness of our method. The code will be available at https://github.com/cong-wu/MCMT-Net.

Robust Representation Learning for Power System Short-Term Voltage Stability Assessment Under Diverse Data Loss Conditions.

With the help of neural network-based representation learning, significant progress has been recently made in data-driven online dynamic stability assessment (DSA) of complex electric power systems. However, without sufficient attention to diverse data loss conditions in practice, the existing data-driven DSA solutions' performance could be largely degraded due to practical defective input data. To address this problem, this work develops a robust representation learning approach to enhance DSA performance against multiple input data loss conditions in practice. Specifically, focusing on the short-term voltage stability (SVS) issue, an ensemble representation learning scheme (ERLS) is carefully designed to achieve data loss-tolerant online SVS assessment: 1) based on an efficient data masking technique, various missing data conditions are handled and augmented in a unified manner for lossy learning dataset preparation; 2) the emerging spatial-temporal graph convolutional network (STGCN) is leveraged to derive multiple diversified base learners with strong capability in SVS feature learning and representation; and 3) with massive SVS scenarios deeply grouped into a number of clusters, these STGCN-enabled base learners are distinctly assembled for each cluster via multilinear regression (MLR) to realize ensemble SVS assessment. Such a divide-and-conquer ensemble strategy results in highly robust SVS assessment performance when faced with various severe data loss conditions. Numerical tests on the benchmark Nordic test system illustrate the efficacy of the proposed approach.

Deep Reinforcement Learning for Voltage Control and Renewable Accommodation Using Spatial-Temporal Graph Information

Renewable energy resources (RERs) have been increasingly integrated into distribution networks (DNs) for decarbonization. However, the variable nature of RERs introduces uncertainties to DNs, frequently resulting in voltage fluctuations that threaten system security and hamper the further adoption of RERs. To incentivize more RER penetration, we propose a deep reinforcement learning (DRL)-based strategy to dynamically balance the trade-off between voltage fluctuation control and renewable accommodation. To further extract multi-time-scale spatial-temporal (ST) graphical information of a DN, our strategy draws on a multi-grained attention-based spatial-temporal graph convolution network (MG-ASTGCN), consisting of ST attention mechanism and ST convolution to explore the node correlations in the spatial and temporal views. The continuous decision-making process of balancing such a trade-off can be modeled as a Markov decision process optimized by the deep deterministic policy gradient (DDPG) algorithm with the help of the derived ST information. We validate our strategy on the modified IEEE 33, 69, and 118-bus radial distribution systems, with outcomes significantly outperforming the optimization-based benchmarks. Simulations also reveal that our developed MG-ASTGCN can to a great extent accelerate the convergence speed of DDPG and improve its performance in stabilizing node voltage in an RER-rich DN. Moreover, our method improves the DN's robustness in the presence of generator failures.